Resampling-based Model Stability Checks
Resampling-based Stability Checks
= use repeated data resampling to check whether model results are stable under changes in the training data.
Common resampling schemes
- Repeated train/test splits: repeatedly create random train/test splits, fit the model, and compare results across splits
- Cross-validation folds: fit models across CV folds and check whether results are consistent
- Bootstrap samples: repeatedly sample with replacement and estimate variation in the result
- Subsampling: repeatedly sample without replacement, often using only part of the training data
What can be checked
| What you track across resamples | What you are checking |
|---|---|
| model performance, e.g. AUC/RMSE | model performance robustness |
| predictions for the same samples | prediction stability |
| selected variables/features | feature selection stability |
| coefficient sizes/signs | parameter stability |
| feature importance rankings | interpretability stability |
Typical workflow
- Resample the data many times
- Fit the model each time
- Record the quantity of interest
- Summarize its variation across resamples
- Check whether the conclusion is stable enough to trust